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Abstract. We develop a theory of planar, origin-symmetric, con- 
vex domains that are inextensible with respect to lattice cover- 
ing, that is, domains such that augmenting them in any way al- 
lows fewer domains to cover the same area. We show that origin- 
symmetric inextensible domains are exactly the origin-symmetric 
convex domains with a circle of outer billiard triangles. We address 
a conjecture by Genin and Tabachnikov about convex domains, not 
necessarily symmetric, with a circle of outer billiard triangles, and 
show that it follows immediately from a result of L. Fejes Toth 
related to the covering problem. 



In a series of papers from 1946 to 1947, Kurt Mahler developed a the- 
ory of planar, origin-symmetric, star-like domains that are irreducible 
with respect to lattice packing, that is, domains such that taking away 
any piece allows more domains to be packed in the same area [5j [6] . 
In particular, he was interested in the problem of identifying convex 
domains such that every convex domain with lower area may be packed 
at a greater number per unit area than the domain in question. This is 
the subject of Reinhardt's conjecture [7j. In particular, Mahler showed 
that the disk is not such a domain, even if the domains of lower area 
to which we compare it are restricted to a small neighborhood of the 
disk with respect to Hausdorff distance [6] . 

Inspired by the work of Mahler, we develop a theory of planar, origin- 
symmetric, convex domains that are inextensible with respect to lattice 
covering, that is, domains such that adding any piece allows fewer do- 
mains to cover the same area. We find that the inextensible domains 
are simply those with a circle of outer billiard triangles, a family of do- 
mains studied previously by Genin and Tabachnikov [3J. The analogue 
of Reinhardt's conjecture for covering was solved by L. Fejes Toth who 
showed that every origin-symmetric convex domain can cover the plane 
with no larger number per unit area than an ellipse of the same area 
[8]. In other words, the ellipse covers the plane with the least efficiency. 
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Genin and Tabachnikov conjecture that out of convex domains, not 
necessarily symmetric, with a circle of critical triangles of a fixed area, 
the ellipse provides the upper bound for the area of the domain [3]. A 
theorem of Fejes Toth which he uses to prove that the ellipse is the 
lease efficient covering shape yields this conjecture immediately. 

We call A a symmetric convex domain (below just "domain"), if 
A is a convex compact subset of M 2 such that K = —A. A lattice 
A = BZ 2 is the image of the integer lattice Z 2 under a nonsingular 
linear map B. The determinant d(A) of a lattice A = B7? is given 
by | det(.B)| and is independent of the basis B used. We will use the 
Hausdorff metric as a distance between domains: S(K,K') = min{e : 
K C K' + eD, K' C K + eD}, where D is the unit disk. As the 
distance between lattices we use the distance between the closest two 
bases: S(A, A') = min{||£ - B'\\ : A = BZ 2 , A = B'Z 2 }, where || • || is 
the Hilbert-Schmidt norm. 

The lattice A is called A-covering if A + A = M 2 . Of all A-covering 
lattices there is at least one lattice A c such that d(A) < d(A c ) whenever 
A is A-covering jl]. We call such a lattice a critical lattice for A and 
its determinant is called the critical determinant of A and denoted 
A (A). Whenever K' D A, any critical lattice of A is also A'-covering, 
and therefore A(A') > A(A). We say that A is extensible if there 
is a domain K' containing A but different from it that has the same 
critical determinant as A. Otherwise, we say A is inextensible. 

Let us begin with an example of an inextensible domain: a parallel- 
ogram. The critical determinant of a parallelogram is equal to its area. 
Let A be a parallelogram and let A' be a proper superdomain. By 
definition there must be a point x G K' \ K. Let K" = conv(A, ±x). 
Since K" is either a parallelogram of a hexagon, its critical determinant 
is equal to its area and therefore A(A) < A(K") < A(K'). 

Bambah and Rogers have showed that if T is a triangle inscribed in 
A, maximizing the area among all triangles inscribed in A, then the 
critical determinant of a domain A is equal to twice the area of T [1] . 
We call such a triangle a critical triangle of A. Consequently any lat- 
tice which is critical for the (possibly degenerate) hexagon conv(T, — T) 
is critical for A. Any critical triangle T must include the origin, other- 
wise a reflection of one of its vertices through the opposite side yields a 
triangle of equal area but with one vertex in the interior of A. When A 
is a parallelogram, T may have the origin on its perimeter. When A is 
not a parallelogram, the origin must be in the interior of T, since oth- 
erwise the hexagon conv(T, — T) degenerates to a parallelogram which 
is simultaneously a proper subdomain of A and of equal critical deter- 
minant, contradicting the inextensibility of parallelograms. Similarly, 
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the hexagon conv(T, — T) must also not degenerate to a parallelogram 
by having one of the vertices of T lie on a side of —T and therefore the 
relative interiors of all sides of T must be interior to K. 

Let vlq = (cos 9, sin 9) be a point on the unit circle at angle 9 from 
the x-axis. The support height of a domain K in the direction ug 
is given by h{9) = max xe ^(x, ug). The support line L(9) is the line 
{x 6 I 2 : (x, ug) = h(9)}, and it intersects the boundary of K but not 
its interior. 

Lemma 1. Let L(9) be a support line of K. Consider all triangles xyz 
inscribed in K in such a way that x G L{9), and the side yz is parallel 
to L(9) (for definiteness, let xyz be arranged counter-clockwise). There 
are points yo and z such that a triangle of the type described maximizes 
the area amongst all such triangles if and only if y = yo and z = zq. 

Proof. Let xyz and x'yV be two triangles maximizing the area. First 
note that the area does not depend on the point lying on L(9). There- 
fore, we may take x = x' without loss of generality. If the altitudes 
taken from x of the two triangles are not equal, then convexity guaran- 
tees that any inscribed triangle of the type described whose altitude is 
between the two altitudes must have strictly greater area. Therefore, 
the altitudes must be equal, and consequently also the length of the side 
opposite x. If the sides opposite x are not identical, then their convex 
hull gives the side of an inscribed triangle of the type described with 
greater area. Therefore, the sides opposite x must be identical. □ 

We denote by A{9) the maximal area of a triangle of the type de- 
scribed in Lemma dj and call a triangle achieving this triangle 
anchored at L(9) (or simply at angle 9). Clearly, the set of all trian- 
gles anchored at support lines of K contains all the critical triangles 
of K and A(K) = 2 maxo<e<27r A(0) = 2v4 max . Note that any critical 
triangle occurs as a triangle anchored at three angles (parallel to its 
three sides) and its symmetric image occurs as a triangle anchored at 
three more angles. We label these angles 9^ % = 1, ... ,6, such that 
9\ < 9 2 < . . . < 9 6 < 9i + 2n and 6^+3 = 9; t + it (the label i is un- 
derstood to extend cyclically such that 9 i+Q = 9i + 2n). Therefore, 
we associate each critical triangle paired with its symmetric image (for 
short, critical triangle pair) with the corresponding angles 9\, ... 9q at 
which they are anchored. We now show that the angles associated with 
two critical triangle pairs intersperse. 

Lemma 2. Let 9i,9s, and 9$ and 9[,9' 3 , and 9' 5 be the angles at which 
two distinct critical triangle of K arise, then there is some even integer 
k such that 9 1 < 9' < 9 3 < 9' < 9 5 < 9' < 9 1 + 2tt. 
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Proof. First let us reduce the problem to the case where all six angles 
and all six triangle vertices are distinct. If the two triangles are an- 
chored at a common support line, then by Lemma [I] they also have two 
vertices in common, and the two remaining vertices lie on a line parallel 
to the line connecting the shared vertices. The lemma in this case then 
follows easily. Therefore, assume that all six angles are distinct. If the 
triangles have a vertex in common, then by convexity of K and by the 
fact that the interior of the triangles must intersect (both must con- 
tain the origin) we conclude that the sides opposite the shared vertex 
must intersect. It follows easily that starting from the shared vertex 
and traveling counterclockwise along the boundary of K, we meet the 
vertices of the two triangles alternatingly and the lemma holds. We 
assume therefore that the six angles are distinct as are the six vertices. 
Furthermore, if K is a parallelogram then the lemma follows easily, so 
we may also assume that K is not a parallelogram and there is no line 
segment in the boundary of K containing three of the six vertices. 

For a proof by contradiction, assume that one of the three intervals 
(0i)03)> (03)05)) an d (05,0i + 27r) includes at least two of the angles 
9[, 9' 3 , and 9' 5 . Since the two triangles must intersect, it includes exactly 
two. Without loss of generality then, we may take 9i < 9[ < 9' 3 < 9% < 
05 < 05 < 0i + 27T. Denote by x, y, and z the vertices incident on the 
support lines L(9i),L(9 3 ), and L{9^) respectively, and similarly denote 
by x',y', and z' the vertices of the other critical triangle. 

We make use of the fact that if ABCDE is a non-degenerate convex 
pentagon, and axea(ABD) < &rea,(ABE) then area(ACD) < a,rea.(ACE) 
This fact is proved, for example, in Ref. [2] under the name of the 
Pentagon Lemma. Using the Pentagon Lemma and the fact that 
area(x'yz) < area(xyz) we have that area(x'yz') < area(xyz'). On 
the other hand, the fact that area(xy'z') < area(x'y'z') gives through 
the Pentagon Lemma that area(xyz') < area(x'yz'). The two inequal- 
ities contradict. □ 

Corollary 1. The angles at which two distinct critical triangle pairs 
are anchored may be labeled in such a way that 9\ < 9[ < 92 < 9' 2 < 
3 < • • • < 6 < 06 < 0i + 2vr. 

Intuitively, a domain can only be inextensible if it has a one param- 
eter family of critical lattices so that any augmentation of the domain 
would allow one of the critical lattices to increase in determinant. We 
prove this now. 

Theorem 1. K is inextensible if and only if A{9) is constant. 
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Proof. First assume that K is extensible. Then there exists a proper 
superdomain K' D K such that maxA'(0) = max A{9). There exists 
a support line L(6 ) of K which intersects the interior of K', and con- 
sequently A(0 Q ) < A'(9 Q ) < max A'(9) = max A(9). Therefore A{9) is 
not constant. 

For the converse assume that A{9) is not constant. Note that A{9) 
must be continuous in 9, so the set of angles such that A{9) = A maiX 
is closed. Consider those support lines L{9) of K that are not tangent 
to K on either side of their intersection with the boundary, that is, 
support lines such that support lines for sufficiently nearby angles have 
the same intersection with the boundary. Call support lines of this 
type non-tangent and call all other support lines tangent. Assume 
that there is an angle 9 such that L(9) is tangent and A{9) < A max . 
If such an angle exists, then we may pick it in such a way that all 
sufficiently nearby angles also have tangent support lines. We can 
then construct superdomains K' e D K with the same support lines as 
K for angles outside an arbitrarily small neighborhood of 9 and such 
that the support lines for angles inside the neighborhood are moved by 
arbitrarily small distances. It is clear that A(K' e ) = A (if) for small 
enough e and K is extensible. Otherwise, there must be for all small 
enough e a critical triangle of K' £ of area greater than A max and with 
a vertex on the part of the boundary of K' £ that is not shared with 
K. The existence of these triangles would imply the existence of an 
anchored triangle for K anchored at L(9) with area at least ^4max; and 
is therefore impossible. 

Suppose on the other hand that all angles for which A(9) is submaxi- 
mal have support lines that are non-tangent. Let us consider an angle 9 
for which A(9) is submaximal and denote as x the point at which L(9) 
intersects dK. By continuity, 9 must be part of an open interval (9\, 9[) 
such that A{9 X ) = A(8[) = A max and A{9') < A max for all 9' e (9x,9[). 
Since all angles 9' G (9i,9[) are submaximal, their support lines are 
non-tangent, and therefore both L(9\) and L(9[) contain x e dK, and 
x is a vertex of two critical triangles anchored at L(9i) and L(9[). De- 
note the other two angles anchoring each of these triangles as #3 and 6* 5 
and 9' 3 and 9' b . Consider the intervals (6*3, #3) and (#5, 9' 5 ). By the inter- 
spersing property (Lemma [2]), A{9) is submaximal at all angles of these 
intervals, therefore all angles of these intervals must have non-tangent 
support lines. It follows that all vertices of the two critical triangles 
coincide. Since this is a contradiction, the case where all submaximal 
angles have non-tangent support lines is impossible. □ 
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Figure 1 . An example of an inextensible domain (left) . 
The boundary of the domain is composed of 4 line seg- 
ments and 12 arcs belonging to 4 ellipses. The three arcs 
belonging to each ellipse are traced out by the vertices of 
critical triangles of that ellipse (right). This construction 
yields a one-parameter family of inextensible domains in- 
terpolating between the disk and the square. 

Genin and Tabachnikov have observed that convex domains, not nec- 
essarily symmetric, such that each support line anchors a critical tri- 
angle can be characterized by the equivalent property that each point 
on its boundary lies on an outer billiard triangle, that is, a circum- 
scribed triangle such that the midpoint of each side is on the boundary 
of the domain [3] . For terseness, we will say that a domain (again, not 
necessary symmetric) that possesses these equivalent properties has a 
circle of critical triangles. The theorem identifies the family of inexten- 
sible symmetric convex domains with the family of symmetric convex 
domains with a circle of critical triangles. Examples of inextensible 
domains include, in addition to the parallelogram and ellipse, the reg- 
ular 6n + 4-gons. Another example of a family of inextensible domains 
interpolating between the disk and the square is given in Figure [TJ 

Genin and Tabachnikov conjecture that, among convex domains with 
a circle of critical triangles of a given area, the ellipse maximizes the 
area of the domain. This in fact follows easily from the following the- 
orem of L. Fejes Toth: 

Theorem 2. (L. Fejes Toth [8]^ Let K be a convex domain and T n the 
inscribed n-gon of maximal area in K , then 
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with equality for, and only for, an ellipse. 
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Corollary 2. Let K be a convex domain with a circle of critical tri- 
angles of area A, then area A' < AixA/\f¥[, with equality for, and only 
for, an ellipse. 
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